Image adjustment apparatus, image adjustment method, and program

ABSTRACT

An image adjustment device includes: an illumination component derivation unit that derives an illumination component of a grayscale image; a reflectance component derivation unit that derives a reflectance component image that is a resulting image in which the illumination component is removed from the grayscale image; a contrast component derivation unit that derives a contrast component based on a contrast value between a pixel of the reflectance component image and a peripheral area of the pixel; a histogram derivation unit that derives a luminance histogram of the grayscale image weighted according to the contrast value for each pixel of the contrast component; a conversion function derivation unit that derives a luminance conversion function for converting a luminance such that a luminance histogram of a converted grayscale image in which the grayscale image is converted by the luminance conversion function and a predetermined histogram are matched with or similar to each other; and a luminance conversion unit that generates the converted grayscale image.

TECHNICAL FIELD

The present invention relates to an image adjustment device, an imageadjustment method, and a program.

BACKGROUND ART

As an effective way to reveal hidden details in an image, there is knowna method of enhancing the contrast. Although various types of imageediting software are available on the market, they require expertise inimage adjustment and a lot of manual work, and accordingly, it isnecessary to implement automatic, image enhancement for various types ofinput images.

Methods of automatically enhancing an image can be roughly classifiedinto two methods, model-based and learning-based. Model-based methodsare attracting more attention than learning-based methods because theydo not rely on labeled training data.

In image enhancement processing, histogram equalization has attractedthe most attention due to its intuitive implementation quality and highefficiency, while it has a problem of low power of discriminatingenhanced contrast.

This occurs because a contrast component is extremely enhanced in anarea with many pixels in an image, such as a background without texture,even if the area is not visually important (meaningless area).

In order to deal with such a problem, there is a method of incorporatingthe spatial information of an image into processing for equalizing theluminance histogram. For example, according to NPD 1, in order to avoidthat background noise are extremely enhanced in an input image,luminance gradients are calculated as local luminance contrast values,and a histogram weighted by the contrast values is reconstructed andthen equal j zed. However, in an image in which illumination componentsare non-uniform, the spatial information of the image exhibits a smallvalue. Therefore, in the method disclosed in NPD 1, there is a problemthat it is not effective to enhance the contrast of an image includingtoo dark areas or too bright areas, which has non-uniform illuminationcomponents.

Further, as described in NPD 2, there is proposed a method in which anatural image is regarded as a combination of an illumination componentimage and a reflectance component image, and an enhanced image isobtained by estimating and removing the illumination components thatprovide global contrast. However, there is a problem that the brightnessis extremely enhanced because the global contrast is completelyeliminated.

Further, NPD 3 proposes a method of enhancing the illuminationcomponents estimated through a bright-pass filter by histogramequalization and recombining the resulting illumination components withthe reflectance components. However, there is a problem that thehistogram equalization results in extremely suppressed contrast in anarea with few pixels in an image even if the area is visually important(meaningful area).

CITATION LIST Non Patent Document

-   [NPD 1] Xiaomeng Wu, Xinhao Liu, Kaoru Hiramatsu, and Kunio Kashino,    “CONTRAST-ACCUMULATED HISTOGRAM EQUALIZATION FOR IMAGE ENHANCEMENT,”    2017 IEEE International Conference on Image Processing (ICIP).-   [NPD 2] Xiaojie Guo, Yu Li, and Haibin Ling, “LIME: Low-Light Image    Enhancement via Illumination Map Estimation,” IEEE TRANSACTIONS ON    IMAGE PROCESSING, VOL. 26, NO. 2, FEBRUARY 2017.-   [NPD 3] Shuhang Wang and Gang Luo, “Naturalness Preserved Image    Enhancement Using a Priori Multi-Layer Lightness Statistics,” IEEE    TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 2, FEBRUARY 2018.

SUMMARY OF THE INVENTION Technical Problem

As described above, in the histogram equalization, the contrast may beextremely enhanced in an area with many pixels in an image even if thearea is not visually important. On the other hand, the method of NonPatent Document (NPD) 1 which is to improve such a problem mayunderestimate the value of the spatial information of an image includingtoo dark areas or the too bright areas; the method of NPD 2 mayextremely enhance the brightness because the global contrast componentis completely eliminated; and the method of NPD 3 may extremely suppressthe contrast in an area with few pixels.

In view of the above circumstances, an object of the present inventionis to provide an adjustment device, an image adjustment method, and aprogram which make it possible to automatically and adaptively enhancethe contrast of a visually important area in an image without extremelyenhancing the contrast and brightness in the image.

Means for Solving the Problem

One aspect of the present invention is an image adjustment deviceincluding: an illumination component derivation unit that derives anillumination component of a grayscale image; a reflectance componentderivation unit that derives a reflectance component image that is aresulting image in which the illumination component is removed from thegrayscale image; a contrast component derivation unit that derives acontrast component based on a contrast value between a pixel of thereflectance component image and a peripheral area of the pixel; ahistogram derivation unit that derives a luminance histogram of thegrayscale image weighted according to the contrast value for each pixelof the contrast component; a conversion function derivation unit thatderives a luminance conversion function for converting a luminance suchthat a luminance histogram of a converted grayscale image in which thegrayscale image is converted by the luminance conversion function and apredetermined histogram are matched with or similar to each other; and aluminance conversion unit that generates the converted grayscale image.

Effects of the Invention

According to the present invention, it is possible to automatically andadaptively enhance the contrast of a visually important area in an imagewithout extremely enhancing the contrast and brightness in the image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a structural example of an imageadjustment device in an embodiment.

FIG. 2 is a diagram illustrating a hardware configuration example of theimage adjustment device in the embodiment.

FIG. 3 illustrates an example of a luminance histogram of a grayscaleimage weighted according to a contrast component in the embodiment.

FIG. 4 illustrates an example of a luminance conversion function in theembodiment.

FIG. 5 is a flowchart illustrating an operation example of the imageadjustment device in the embodiment.

FIG. 6 illustrates an example of images in the embodiment.

FIG. 7 illustrates a first example of a set of an enhanced input imageand images obtained by other methods in the embodiment.

FIG. 8 illustrates a second example of a set of an enhanced input imageand images obtained by other methods in the embodiment.

FIG. 9 illustrates a third example of a set of an enhanced input imageand images obtained by other methods in the embodiment.

FIG. 10 illustrates a fourth example of a set of an enhanced input imageand images obtained by other methods in the embodiment.

FIG. 11 illustrates a fifth example of a set of an enhanced input imageand images obtained by other methods in the embodiment.

FIG. 12 illustrates a sixth example of a set of an enhanced input imageand images obtained by other methods in the embodiment.

FIG. 13 illustrates a seventh example of a set of an enhanced inputimage and images obtained by other methods in the embodiment.

FIG. 14 illustrates an eighth example of a set of an enhanced inputimage and images obtained by other methods in the embodiment.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention will be described in detail withreference to the drawings.

FIG. 1 is a diagram illustrating a configuration example of an imageadjustment device 1. The image adjustment device 1 (image adjustmentapparatus) is a device for adjusting an image. For example, the imageadjustment device 1 enhances the contrast of an image. The imageadjustment device 1 may adjust the brightness of an image. For example,the image adjustment device 1 adjusts the contrast and brightness ofareas that are visually important in an image and does not adjust thecontrast and brightness of areas that are not visually important in theimage so much.

The image adjustment device 1 includes an input unit 10, a grayscaleimage derivation unit 11, an illumination component derivation unit 12,a reflectance component derivation unit 13, a contrast componentderivation unit 14, a histogram derivation unit 15, a conversionfunction derivation unit 16, a luminance conversion unit 17, areflectance component addition unit 18, an image reconstruction unit 19,and an output unit 20. Note that the image adjustment device 1 does nothave to include the reflectance component addition unit 18.

FIG. 2 is a diagram illustrating a hardware configuration example of theimage adjustment device 1. The image adjustment device 1 includes aprocessor 2, a memory 3, and a storage device 4.

Part or all of the respective functional units of the image adjustmentdevice 1 are implemented as software by the processor 2 such as a CPU(Central Processing Unit) executing a program loaded into the memory 3from the storage device 4 which is a non-volatile storage medium(non-transitory storage medium). The program may be recorded on acomputer-readable storage medium. The computer-readable storage mediumis a non-transitory storage medium, including a portable medium, such asa flexible disk, magneto optical disk, ROM (Read Only Memory), andCD-ROM (Compact Disc Read Only Memory), and a storage device, such as ahard disk built in a computer system, for example. The program may bereceived via a telecommunication line.

Part or all of the respective functional units of the image adjustmentdevice 1 may be implemented using hardware including, for example, anelectronic circuit or circuitry using an LSI (Large Scale Integrationcircuit), ASIC (Application Specific Integrated Circuit), PLD(Programmable Logic Device), or FPGA (Field Programmable Gate Array).

Returning to FIG. 1 , the outline of the image adjustment device 1 willbe described.

The input unit 10 acquires a color image or a grayscale image as aninput image “C_(in)”. The grayscale image is an image representing thevalue or luminance of the pixels of the input image. The input unit 10outputs the input image to the grayscale image derivation unit 11 andthe image reconstruction unit 19. In the following, matrix operationsare performed element by element.

For the input image being a color image, the grayscale image derivationunit 11 derives a grayscale image from the color image. The grayscaleimage derivation unit 11 may acquire a grayscale image from the inputunit 10 instead of deriving the grayscale image. The grayscale imagederivation unit 11 outputs the grayscale image to the illuminationcomponent derivation unit 12, the reflectance component derivation unit13, the histogram derivation unit 15, the luminance conversion unit 17,and the image reconstruction unit 19.

The illumination component derivation unit 12 equalizes the grayscaleimage. The illumination component derivation unit 12 generates theresulting image in which the grayscale image is equalized as anillumination component image. The illumination component image is animage representing the illumination component of the grayscale image.The illumination component derivation unit 12 outputs the illuminationcomponent image to the reflectance component derivation unit 13.

The reflectance component derivation unit 13 acquires the illuminationcomponent image from the illumination component derivation unit 12. Thereflectance component derivation unit 13 acquires the grayscale imagefrom the grayscale image derivation unit 11. The reflectance componentderivation unit 13 generates the resulting image in which theillumination component of the illumination component image is removedfrom the grayscale image as a reflectance component image. Thereflectance component derivation unit 13 outputs the reflectancecomponent image to the contrast component derivation unit 14 and thereflectance component addition unit 18.

The contrast component derivation unit 14 derives, for each pixel of thereflectance component image, the value of a local gradient of thereflectance component in the reflectance component image as a weightcoefficient. In other words, the contrast component derivation unit 14derives, for each pixel of the reflectance component image, a contrastvalue between the pixel in the reflectance component image and aperiphery of the pixel as a weight coefficient.

The contrast component derivation unit 14 generates an image havingpixel values each of which is the contrast value between a pixel in thereflectance component image and a periphery of the pixel (hereinafter,referred to as “contrast component”). The contrast component derivationunit 14 outputs the contrast component to the histogram derivation unit15.

The histogram derivation unit 15 acquires the contrast component fromthe contrast component derivation unit 14. The histogram derivation unit15 acquires the grayscale image from the grayscale image derivation unit11. The histogram derivation unit 15 derives a luminance histogram ofthe grayscale image weighted according to the contrast value for eachpixel of the contrast component. The histogram derivation unit 15outputs the luminance histogram to the conversion function derivationunit 16.

The conversion function derivation unit 16 acquires the luminancehistogram from the histogram derivation unit 15. The conversion functionderivation unit 16 acquires a predetermined histogram from the memory 3.The predetermined histogram is a histogram determined in advanceaccording to the specification of enhancement, and is, for example, ahistogram of uniformly distributed luminance. The distribution of thepredetermined histogram may be any probability distribution. Forexample, the distribution of the predetermined histogram may be aprobability distribution given in the form of a logarithmic or linearfunction.

The conversion function derivation unit 16 derives a luminanceconversion function such that the luminance histogram of a grayscaleimage with converted luminance (hereinafter referred to as “convertedgrayscale image”) and the predetermined histogram are matched with orsimilar to each other. As a result, the predetermined histogram and theluminance histogram of the converted grayscale image output from theluminance conversion unit 17 are matched with or similar to each other.The conversion function derivation unit 16 outputs the luminanceconversion function to the luminance conversion unit 17.

The luminance conversion unit 17 acquires the luminance conversionfunction from the conversion function derivation unit 16. The luminanceconversion unit 17 acquires the grayscale image from the grayscale imagederivation unit 11. The luminance conversion unit 17 converts theluminance of the grayscale image by applying the luminance conversionfunction to the luminance of each pixel of the grayscale image. In otherwords, the luminance conversion unit 17 generates a result of applyingthe luminance conversion function to the luminance of each pixel of thegrayscale image as a converted grayscale image. The luminance conversionunit 17 outputs the converted grayscale image to the reflectancecomponent addition unit 18.

The reflectance component addition unit 18 acquires the convertedgrayscale image from the luminance conversion unit 17. The reflectancecomponent addition unit 18 acquires the reflectance component image fromthe reflectance component derivation unit 13. The reflectance componentaddition unit 18 adds the reflectance component of the reflectancecomponent image to the converted grayscale image to generate a grayscaleimage with further enhanced contrast (hereinafter referred to as“enhanced grayscale image”). The reflectance component addition unit 18generates the enhanced grayscale image by synthesizing the convertedgrayscale image and the reflectance component image. The reflectancecomponent addition unit 18 outputs the enhanced grayscale image to theimage reconstruction unit 19.

The image reconstruction unit 19 acquires the enhanced grayscale imagefrom the reflectance component addition unit 18. The imagereconstruction unit 19 acquires the input image from the input unit 10.The image reconstruction unit 19 acquires the grayscale image from thegrayscale image derivation unit 11. The image reconstruction unit 19reconstructs, from the enhanced grayscale image, the color informationof the input image, and the grayscale image, an input image withenhanced contrast (hereinafter, referred to as “enhanced input image”).The image reconstruction unit 19 outputs the enhanced input image to theoutput unit 20. The output unit 20 outputs the enhanced input image to apredetermined external device (not illustrated).

Next, the details of the image adjustment device 1 will be described.

When the grayscale image derivation unit 11 acquires a color image asthe input image from the input unit 10, the grayscale image derivationunit 11 derives the grayscale image of the input image from the inputimage. When the grayscale image derivation unit 11 acquires a grayscaleimage as the input image from the input unit 10, the grayscale imagederivation unit 11 outputs the grayscale image thus input as it is tothe illumination component derivation unit 12, the reflectance componentderivation unit 13, the histogram derivation unit 15, the luminanceconversion unit 17, and the image reconstruction unit 19.

The following is a case where the grayscale image derivation unit 11acquires a color image as an input image from the input unit 10. Thegrayscale image derivation unit 11 derives a grayscale image as an imagerepresenting the maximum value in the color components of the RGB (Red,Green, Blue) component for each pixel of the input image. The maximumvalue in the RGB component is the same as the value component (Vcomponent) of the input image of the HSV color space (color space ofhue, saturation, and value/brightness). The grayscale image derivationunit 11 may derive a grayscale image as an image representing the RGBcomponent for each pixel of the input image.

The grayscale image derivation unit 11 may derive a grayscale image asan image representing the I component (luminance component) of the HSI(hue, saturation, intensity representing the average value of RGBcomponent) color space for each pixel. The grayscale image derivationunit 11 may derive a grayscale image as an image representing the Lcomponent (lightness component) of the HSL (hue, saturation, lightnessrepresenting the average value of the maximum and minimum values of RGBcomponent) color space for each pixel. The grayscale image derivationunit 11 may derive a grayscale image as an image representing the Ycomponent (luminance component) of the XYZ color space or YUV(luminance, blue-luminance difference, and red-luminance difference)color space for each pixel. The grayscale image derivation unit 11 mayderive a grayscale image as an image representing the L component(lightness component) of the LUV color space or the Lab color space foreach pixel.

The illumination component derivation unit 12 equalizes the grayscaleimage to generate the resulting image in which the grayscale image isequalized as an illumination component image. The method by which theillumination component derivation unit 12 equalizes the grayscale imageis not limited to a specific equalization method.

For example, the illumination component derivation unit 12 may generateas an illumination component image the resulting image in which thegrayscale image is equalized by a median filter, a bilateral filter, aguided filter, or an anisotropic diffusion filter. In the following, asan example, the illumination component derivation unit 12 generates asan illumination component image the resulting image in which thegrayscale image is equalized by edge-preserving smoothing. Inedge-preserving smoothing, for each pixel, the ratio of the average ofthe absolute values of the first-order derivatives of luminance to theabsolute of the average of the first-order derivatives of luminance in alocal area around the pixel is measured as a score indicating thestrength of the texture. When the grayscale image is input to theillumination component derivation unit 12, an output image (theresulting image in which the grayscale image is equalized) is optimizedso that the total strength of the texture of the output image isminimized. In the resulting image in which the grayscale image isequalized by edge-preserving smoothing, the halo effect is unlikely tooccur in the reflectance component of the grayscale image.

The reflectance component derivation unit 13 derives a result ofremoving the illumination component from the grayscale image as areflectance component image. The reflectance component is expressed byEquation (1).

$\begin{matrix}\left\lbrack {{Formula}1} \right\rbrack &  \\{R = {\log_{10}\left( \frac{A_{in}}{I} \right)}} & (1)\end{matrix}$

Here, “A_(in)” represents a grayscale image. “I” represents theillumination component of the grayscale image. “R” represents thereflectance component of the grayscale image. According to the Retinextheory, a natural image is a combination of an illumination componentimage and a reflectance component image. The illumination componentrepresents the global contrast within the grayscale image of the naturalimage. The reflectance component represents the details in an areawithin the grayscale image of the natural image.

In Equation (1), the reflectance component is derived by using alogarithmic function. By using the logarithmic function, the differencein the reflectance component in a dark area in an image becomes large.By using the logarithmic function, the difference in the reflectancecomponent in a bright area in the image becomes small. In naturalimages, bright areas such as light sources and the sky are usually lessvisually important. On the other hand, when an image is captured underpoor illumination conditions such as backlight, low key illumination,and underexposure, it is more difficult to enhance important imagedetails hidden in dark areas. Therefore, the use of a logarithmicfunction as in Equation (1) is effective for deriving the reflectancecomponent. Note that, although a common logarithm is used in Equation(1), a logarithm having any real number larger than 1 as the base may beused.

As long as the illumination component is removed from the grayscaleimage, the method for deriving the reflectance component image is notlimited to a specific deriving method. For example, the reflectancecomponent derivation unit 13 may derive a result of dividing each pixelvalue of the grayscale image by each pixel value of the illuminationcomponent as a reflectance component image. Further, for example, thereflectance component derivation unit 13 may derive a result ofsubtracting the illumination component from the grayscale image as areflectance component image.

The contrast component derivation unit 14 derives a contrast value “φ”of the reflectance component between a pixel “q” and a periphery of thepixel for each pixel of the reflectance component image as illustratedin Equation (2).

$\begin{matrix}\left\lbrack {{Formula}2} \right\rbrack &  \\{{\phi(q)} = {\sum\limits_{q^{\prime} \in {N(q)}}{❘{{r\left( \overset{.}{q} \right)} - {r\left( q^{\prime} \right)}}❘}}} & (2)\end{matrix}$

Here, “N(q)” represents a set of coordinates of pixels adjacent to thepixel “q” in the reflectance component image. For example, “N(q)” is thevon Neumann neighborhood, the extended von Neumann neighborhood, or theMoore neighborhood. In the following, the pixel value “{r(q)}” of thepixel “q” in the reflectance component image is expressed as “R”.

In the reflectance component image, the images of visually importantobjects may be different in size from each other. In this respect, thecontrast component derivation unit 14 uses a multiresolution method toderive contrast values at different resolutions from each other so thatthe images of all visually important objects are represented using thecontrast values. For example, the contrast component derivation unit 14generates a plurality of reflectance components “R₁, R₂, . . . , R_(L)”at different resolutions from each other. “L” is the number ofreflectance components to be generated. In the following, “L” is 4 as anexample. The resolution of the reflectance component “R₁(=R)” is thelargest among the resolutions of the plurality of reflectancecomponents. The resolution of the reflectance component “R_(L)(=R)” isthe smallest among the resolutions of the plurality of reflectancecomponents.

The contrast component derivation unit 14 derives the reflectancecomponent “R₂” by downsampling the reflectance component “R₁(=R)” inhalf using the bicubic interpolation method. The contrast componentderivation unit 14 repeats the downsampling (change of resolution) “L−1”times until the reflectance component “R_(L)” is obtained.

The contrast component derivation unit 14 derives the contrast value“φ(q)” as in Equation (2) based on each downsampled reflectancecomponent. The contrast component derivation unit 14 upsamples thecontrast values “φ(q)” of all the downsampled reflectance components tothe same resolution as the reflectance component “R”, and then uses thegeometric mean to synthesize the resulting contrast values for eachpixel “q”. The contrast component derivation unit 14 outputs thecontrast component “Φ(q)” (image) synthesized for the pixel “q” to thehistogram derivation unit 15.

The method for deriving the contrast value is not limited to a specificderiving method as long as, for each pixel of the reflectance componentimage, the contrast value between the pixel and the periphery of thepixel is derived.

For example, instead of using a multiresolution method, the contrastcomponent derivation unit 14 may output to the histogram derivation unit15 a contrast component whose pixel values are contrast values derivedonly based on the reflectance component “R” based on Equation (2).

For example, instead of using a multiresolution method, the contrastcomponent derivation unit 14 may output to the histogram derivation unit15 a result of applying an edge detection filter such as a sobel filterto the reflectance component image, as a contrast component whose pixelvalues are contrast values.

The histogram derivation unit 15 derives a histogram of the luminance ofthe grayscale image weighted according to the contrast values of thereflectance component image. The luminance histogram of a grayscaleimage “A_(in)={a (q)}” is expressed by Equation (3). Accordingly, theprobability of occurrence “p_(a)(k)” of a pixel luminance value “k∈[0,K)” in the grayscale image “A_(in)” is expressed by Equation (3). Here,“a (q)∈[0, K)” represents the luminance of each pixel “q”. “K” is thetotal number of luminance values in the grayscale image. In thefollowing, “K” is 256 as an example.

$\begin{matrix}\left\lbrack {{Formula}3} \right\rbrack &  \\{{p_{a}(k)} = \frac{\sum_{q}{{\Phi(q)}{\delta\left( {{a(q)},k} \right)}}}{\sum_{q}{\Phi(q)}}} & (3)\end{matrix}$

Here, “Φ(q)” represents a contrast component synthesized for the pixel“q”. “δ(·,·)” is the Kronecker delta.

Equation (3) represents that each pixel of the grayscale imageadaptively contributes to the derivation of the histogram. Theprobability of occurrence “p_(a)(k)” represents an expected value of thecontrast component corresponding to the luminance value “k” of a pixelof the grayscale image.

FIG. 3 illustrates an example of a luminance histogram of a grayscaleimage weighted according to a contrast component. The line indicated by“HE (histogram equalization)” in FIG. 3 represents a histogramrepresenting the frequency of appearance of the pixel values (luminance)of a grayscale image for each luminance. The line indicated by “CACHE”(Contrast-ACcumulated Histogram Equalization) in FIG. 3 represents ahistogram of the luminance of the grayscale image weighted according tothe luminance gradients as disclosed in NPD 1. The line indicated by“Proposed method” in FIG. 3 represents a histogram of the luminance ofthe grayscale image weighted according to the contrast values of thereflectance component image.

The conversion function derivation unit 16 derives a luminanceconversion function such that the luminance histogram of the convertedgrayscale image is matched with or similar to a predetermined histogram.The format of a luminance conversion function “T(k)” is “b(q)=T(a(q))”.The conversion function derivation unit 16 derives the luminanceconversion function “T(k)” for the luminance conversion unit 17 togenerate a converted grayscale image “B={b(q)}” as in Equation (4).

$\begin{matrix}\left\lbrack {{Formula}4} \right\rbrack &  \\\begin{matrix}{{T(k)} = {\arg\min\limits_{T}{❘{{P_{b}\left( {T(k)} \right)} - {P_{a}(k)}}❘}}} \\{= {P_{b}^{- l}\left( {P_{a}(k)} \right)}}\end{matrix} & (4)\end{matrix}$

Here, “P_(a)(k)” represents a cumulative distribution functioncorresponding to the histogram of the luminance (probability ofoccurrence) “p_(a)(k)” of the grayscale image “A_(in)”. “P_(b)(k)”represents a cumulative distribution function corresponding to thehistogram of the luminance (probability of occurrence) “p_(b)(k)” of theconverted grayscale image “B”.

In the following, the predetermined histogram is a histogram ofuniformly distributed luminance as an example. The conversion functionderivation unit 16 derives a luminance conversion function such that thehistogram of uniformly distributed luminance and the luminance histogramof the converted grayscale image are matched with or similar to eachother. Specifically, the conversion function derivation unit 16 derivesa luminance conversion function such that the histogram of uniformlydistributed luminance and the histogram of the luminance (probability ofoccurrence) “p_(b)(k)” of the converted grayscale image “B” are matchedwith or similar to each other. The processing for the conversionfunction derivation unit 16 to derive the luminance conversion functionis processing corresponding to equalization of the luminance histogram.Equation (4) is expressed as Equation (5).

[Formula 5]

T(k)=(K−1)P _(a)(k)  (5)

FIG. 4 is a diagram illustrating an example of a luminance conversionfunction for equalizing the luminance histogram illustrated in FIG. 3 .The horizontal axis represents the luminance (input luminance) of anon-equalized histogram. The vertical axis represents the luminance(output luminance) of an equalized histogram. An increase of the outputluminance with respect to the input luminance is proportional to theexpected value “p_(a) (k)” of the contrast component corresponding tothe luminance value “k” of a pixel of the grayscale image.

The luminance conversion unit 17 derives the luminance of each pixel ofthe converted grayscale images “B={b(q)}” based on the luminanceconversion functions “T(k)” and “b(q)=T(a(q))”. The luminance conversionunit 17 outputs to the reflectance component addition unit 18 a resultof applying the luminance conversion function to the luminance of eachpixel of the grayscale image, as the converted grayscale image“B={b(q)}”.

The reflectance component addition unit 18 generates an enhancedgrayscale image “A_(out)” by adding the reflectance component to theconverted grayscale image “B={b(q)}”. The local contrast inherited fromthe grayscale image “Ai,” is already included in the converted grayscaleimage “B”. Accordingly, the details of the grayscale image are enhancedby adding the reflectance component to the converted grayscale imageeven when the reflectance component is maintained or reduced.

The enhanced grayscale image output from the reflectance componentaddition unit 18 is represented as “A_(out)=B+eR”. Here, “e” is ahyperparameter. The hyperparameter “e” is any real number, for example,a real number selected from a range of [0, 1]. In the following, “e” is0.5 as an example.

The image reconstruction unit 19 reconstructs an enhanced input image(color image with enhanced contrast) by using the input image, thegrayscale image, and the enhanced grayscale image “A_(out)”.Accordingly, the enhanced input image “C_(out)” is a color image(enhanced color image) into which the enhanced grayscale image “A_(out)”has been converted based on the input image and the grayscale image. Theenhanced input image “C_(out)” is represented by Equation (6).

$\begin{matrix}\left\lbrack {{Formula}6} \right\rbrack &  \\{C_{out} = {\left( \frac{C_{in}}{A_{in}} \right)^{\alpha}{A_{out}.}}} & (6)\end{matrix}$

Here, “α” is a hyperparameter. The hyperparameter “α” is 1 or a realnumber close to 1. In the following, “α” is 1 as an example.

Even in a case where the grayscale image derivation unit 11 derives agrayscale image by using the I component of the HSI color space or the Lcomponent of the HSL color space instead of the grayscale imagederivation unit 11 deriving a grayscale image by using the V componentof the HSV color space, the image reconstruction unit 19 generates anenhanced input image by using Equation (6).

In a case where the grayscale image derivation unit 11 derives agrayscale image by using the Y component of the XYZ color space or theYUV color space, the image reconstruction unit 19 may acquire thegrayscale image from the grayscale image derivation unit 11. The imagereconstruction unit 19 may replace the Y component in the enhanced inputimage with an enhanced grayscale image.

In a case where the grayscale image derivation unit 11 derives agrayscale image by using the L component of the LUV color space or theLAB color space, the image reconstruction unit 19 may acquire thegrayscale image from the grayscale image derivation unit 11. The imagereconstruction unit 19 may replace the L component in the enhanced inputimage with an enhanced grayscale image.

In a case where the grayscale image derivation unit 11 derives threegrayscale images by using the respective color components of the RGBcomponents of the input image, the image reconstruction unit 19 maygenerate an enhanced input image by synthesizing the three grayscaleimages.

Next, an operation example of the image adjustment device 1 will bedescribed.

FIG. 5 is a flowchart illustrating the operation example of the imageadjustment device 1. The grayscale image derivation unit 11 acquires theinput image from the input unit 10 (step S101). The grayscale imagederivation unit 11 generates a grayscale image of the input image (stepS102). The illumination component derivation unit 12 generates anillumination component image based on the grayscale image (step S103).The reflectance component derivation unit 13 generates a reflectancecomponent image based on the grayscale image and the illuminationcomponent image (step S104).

The contrast component derivation unit 14 generates a contrast componentbased on the reflectance component image (step S105). The histogramderivation unit 15 derives a luminance histogram based on the grayscaleimage and the contrast component (step S106). The conversion functionderivation unit 16 derives a luminance conversion function such that theluminance histogram of the converted grayscale image and a predeterminedhistogram are matched with or similar to each other (step S107).

The luminance conversion unit 17 generates a converted grayscale imagebased on the grayscale image and the luminance conversion function (stepS108). The reflectance component addition unit 18 generates an enhancedgrayscale image based on the converted grayscale image and thereflectance component image (step S109). The image reconstruction unit19 generates an enhanced input image based on the enhanced grayscaleimage, the input image, and the grayscale image (step S110). The outputunit 20 outputs the enhanced input image to a predetermined externaldevice (step S111).

As described above, the image adjustment device 1 of the embodimentincludes the illumination component derivation unit 12, the reflectancecomponent derivation unit 13, the contrast component derivation unit 14,the histogram derivation unit 15, the conversion function derivationunit 16, and the luminance conversion unit 17. The illuminationcomponent derivation unit 12 derives an illumination component of agrayscale image. The reflectance component derivation unit 13 derives areflectance component image which is a resulting image in which theillumination component is removed from the grayscale image. The contrastcomponent derivation unit 14 derives a contrast component based on acontrast value between a pixel of the reflectance component image and aperipheral area of the pixel. The histogram derivation unit 15 derives aluminance histogram of the grayscale image weighted according to thecontrast value for each pixel of the contrast component. The conversionfunction derivation unit 16 derives a luminance conversion function suchthat the luminance histogram of the converted grayscale image and apredetermined histogram are matched with or similar to each other. Theluminance conversion unit 17 generates a converted grayscale image byusing the luminance conversion function. The image adjustment device 1of the embodiment may include the reflectance component addition unit 18and the image reconstruction unit 19. The reflectance component additionunit 18 may generate an enhanced grayscale image which is a resultingimage in which the converted grayscale image and the reflectancecomponent image are synthesized. The image reconstruction unit 19converts the enhanced input image into an image having color informationbased on the color image (input image) and the grayscale image.

In this way, the image adjustment device 1 removes the illuminationcomponent from the input image (grayscale image) before deriving localluminance gradients, so that contrast values in local dark areas can beaccurately derived. This makes it possible to enhance, according to theinput image (grayscale image), the contrast of visually important areasin the image without extremely enhancing the contrast and brightness inthe image.

In other words, the image adjustment device 1 removes the illuminationcomponent from the input image before deriving local contrast values inthe input image. As a result of removing the illumination component fromthe input image, the reflectance component of the input image remains.This reflectance component does not contain any illumination component,and therefore is completely unaffected by illumination appearing in thecaptured input image. As a result, the image adjustment device 1 canaccurately derive local contrast values in dark area of the input image(grayscale image).

FIG. 6 is a diagram illustrating an example of various images. FIG. 6illustrates an input image 100, an illumination component image 101, areflectance component image 102, a contrast component 103, a convertedgrayscale image 104, an enhanced grayscale image 105, a histogramequalized image 110, a CACHE image 120, and a CACHE contrast component121.

The input image 100 is an example of the input image. The input image100 includes images of one or more objects (subjects). The images ofobjects in the input image 100 are, for example, an image of a flower,an image of a bottle, and an image of a painting.

The illumination component image 101 is an example of the illuminationcomponent image derived by the grayscale image derivation unit 11. Thereflectance component image 102 is an example of the reflectancecomponent image derived by the illumination component derivation unit12. The contrast component 103 is an example of the image derived by thecontrast component derivation unit 14. The contrast component 103represents the contrast (spatial information) in local dark areas inmore detail than the CACHE contrast component 121. The images ofvisually important areas in the image (the images of the objects) havecontrast values of many reflectance components as compared to images ofareas that are not visually important (images other than the objects).

The converted grayscale image 104 is an example of the image derived bythe luminance conversion unit 17. In the converted grayscale image 104,the brightness of the background is improved as compared with thehistogram equalized image 110 and the CACHE image 120. The enhancedgrayscale image 105 is an example of the image derived by thereflectance component addition unit 18.

The histogram equalized image 110 is an image representing a result ofequalizing the luminance histogram of the input image 100.

The CACHE image 120 is an image generated based on local luminancecontrasts (luminance gradients) in the grayscale image (see NPD 1). TheCACHE contrast component 121 is an image representing luminancecontrasts (luminance gradients) of the CACHE image 120. Despite theabundance of image details in the input image 100, the CACHE contrastcomponent 121 exhibits low luminance gradients in dark areas. As aresult of enhancing the contrast of the image based on such luminancegradients, in the CACHE image 120, the brightness of the dark areas andthe visibility of the details such as a bouquet are not improved at all.This is because the magnitudes of the local contrast values used asspatial information are significantly reduced by non-uniformillumination (lack of brightness in dark areas).

Various image examples are now presented: an enhanced input image, ahistogram equalized image, an image obtained by the method of NPD 1(CACHE image), an image obtained by the method of NPD 2 (LIME (Low-LightImage Enhancement) image), and an image obtained by the method of NPD 3(NPIE (Naturalness Preserved Image Enhancement) image).

Each of FIG. 7 to FIG. 14 illustrates an example of a set of an inputimage, an enhanced input image, a histogram equalized image, and imagesaccording to the respective methods of NPDs 1 to 3. In FIG. 7 to FIG. 14, the input image 100 and input images 200, 300, 400, 500, 600, 700, and800 are illustrated as examples of input images, respectively.

In the histogram equalized image 110 and histogram equalized images 210,310, 410, 510, 610, 710, and 810, the visibility of the objects in theforeground is reduced, and the contrast of the background area with manypixels is increased. Each histogram equalized image has lowdiscriminating power for visually important areas in the image.

In the CACHE image 120 and CACHE images 220, 320, 420, 520, 620, 720,and 820, the contrast is naturally enhanced, but the brightness in thedark areas and the ability to enhance the local contrasts are limited.

In the LIME image, the reflectance component is assumed to be the idealoutput for contrast enhancement, and the illumination component iscompletely removed from the input image. As a result of completelyremoving the illumination component from the input image, the brightnessis extremely enhanced in each of the LIME image 130 and LIME images 230,330, 430, 530, 630, 730, and 830. This extreme enhancement causes a lossof contrast in the local areas in each LIME image.

In the NPIE image 140 and NPIE images 240, 340, 440, 540, 640, 740, and840, the illumination component is estimated through a bright-passfilter. The illumination component is enhanced using histogramequalization, and the enhanced illumination component and thereflectance component are recombined. In addition, due to the nature ofthe bright-pass, the brightness in each histogram equalized image isinsufficient.

In the enhanced input image 150 and enhanced input images 250, 350, 450,550, 650, 750, and 850, the contrast is well-balanced and naturallyimproved. In each enhanced input image, the brightness in dark areaswhere the local contrasts in the input image cannot be easily observedis improved. Further, in each enhanced input image, extreme enhancementof brightness is suppressed according to the illumination component inthe input image.

Although the embodiment of the present invention has been described indetail above with reference to the drawings, the specific configurationis not limited to such an embodiment, and includes any designs and thelike without departing from the spirit and scope of the presentinvention.

INDUSTRIAL APPLICABILITY

The present invention is applicable to an image processing device.

REFERENCE SIGNS LIST

-   1 Image adjustment device-   2 Processor-   3 Memory-   4 Storage device-   10 Input unit-   11 Grayscale image derivation unit-   12 Illumination component derivation unit-   13 Reflectance component derivation unit-   14 Contrast component derivation unit-   15 Histogram derivation unit-   16 Conversion function derivation unit-   17 Luminance conversion unit-   18 Reflectance component addition unit-   19 Image reconstruction unit-   20 Output unit-   100 Input image-   101 Illumination component image-   102 Reflectance component image-   103 Contrast component-   104 Converted grayscale image-   105 Enhanced grayscale image-   110 Histogram equalized image-   120 CACHE image-   121 CACHE contrast component-   150 Enhanced input image-   200 Input image-   210 Histogram equalized image-   220 CACHE image-   230 LIME image-   240 NPIE image-   250 Enhanced input image-   300 Input image-   310 Histogram equalized image-   320 CACHE image-   330 LIME image-   340 NPIE image-   350 Enhanced input image-   400 Input image-   410 Histogram equalized image-   420 CACHE image-   430 LIME image-   440 NPIE image-   450 Enhanced input image-   500 Input image-   510 Histogram equalized image-   520 CACHE image-   530 LIME image-   540 NPIE image-   550 Enhanced input image-   600 Input image-   610 Histogram equalized image-   620 CACHE image-   630 LIME image-   640 NPIE image-   650 Enhanced input image-   700 Input image-   710 Histogram equalized image-   720 CACHE image-   730 LIME image-   740 NPIE image-   750 Enhanced input image-   800 Input image-   810 Histogram equalized image-   820 CACHE image-   830 LIME image-   840 NPIE image-   850 Enhanced input image

1. An image adjustment device comprising: an illumination componentderivator that derives an illumination component of a grayscale image; areflectance component derivator that derives a reflectance componentimage that is a resulting image in which the illumination component isremoved from the grayscale image; a contrast component derivator thatderives a contrast component based on a contrast value between a pixelof the reflectance component image and a peripheral area of the pixel; ahistogram derivator that derives a luminance histogram of the grayscaleimage weighted according to the contrast value for each pixel of thecontrast component; a conversion function derivator that derives aluminance conversion function for converting a luminance such that aluminance histogram of a converted grayscale image in which thegrayscale image is converted by the luminance conversion function and apredetermined histogram are matched with or similar to each other; and aluminance converter that generates the converted grayscale image,wherein each of the illumination component derivator, the reflectancecomponent derivator, the contrast component derivator, the histogramderivator, the conversion function derivator, and the luminanceconverter is implemented by: i) computer executable instructionsexecuted by at least one processor, ii) at least one circuitry or iii) acombination of computer executable instructions executed by at least oneprocessor and at least one circuitry.
 2. The image adjustment deviceaccording to claim 1, further comprising a reflectance component adderthat generates an enhanced grayscale image which is a resulting image inwhich the converted grayscale image and the reflectance component imageare synthesized, wherein each of the reflectance component adder isimplemented by: i) computer executable instructions executed by at leastone processor, ii) at least on circuitry or iii) a combination ofcomputer executable instructions executed by at least one processor andat least one circuitry.
 3. The image adjustment device according toclaim 2, further comprising: a grayscale image derivator that derivesthe grayscale image from a color image; and an image reconstructor thatconverts the enhanced grayscale image into an image having colorinformation based on the color image and the grayscale image, whereineach of the grayscale image derivator and the image reconstructor isimplemented by: i) computer executable instructions executed by at leastone processor, ii) at least one circuitry or iii) a combination ofcomputer executable instructions executed by at least one processor andat least one circuitry.
 4. The image adjustment device according toclaim 1, wherein the predetermined histogram is a histogram of uniformlydistributed luminance.
 5. An image adjustment method performed by animage adjustment device, the image adjustment method comprising:deriving an illumination component of a grayscale image; deriving areflectance component image that is a resulting image in which theillumination component is removed from the grayscale image; deriving acontrast component based on a contrast value between a pixel of thereflectance component image and a peripheral area of the pixel; derivinga luminance histogram of the grayscale image weighted according to thecontrast value for each pixel of the contrast component; deriving aluminance conversion function for converting a luminance such that aluminance histogram of a converted grayscale image in which thegrayscale image is converted by the luminance conversion function and apredetermined histogram are matched with or similar to each other; andgenerating the converted grayscale image.
 6. A non-transitorycomputer-readable storage medium storing a program for causing acomputer to execute processes, the processes comprising: deriving anillumination component of a grayscale image; deriving a reflectancecomponent image that Is a resulting Image in which the illuminationcomponent is removed from the grayscale Image; deriving a contrastcomponent based on a contrast value between a pixel of the reflectancecomponent image and a peripheral area of the pixel; deriving a luminancehistogram of the grayscale image weighted according to the contrastvalue for each pixel of the contrast component; deriving a luminanceconversion function for converting a luminance such that a luminancehistogram of a converted grayscale image in which the grayscale image isconverted by the luminance conversion junction and a predeterminedhistogram are matched with or similar to each other; and generating theconverted grayscale image.